Repository navigation
MIT-653: show the active store in doctor, status and a SessionStart draft - #178
Merged
Merged
Conversation
…rt draft
`commons doctor` gains a `store` check and `commons status` opens with a
`store:` line: `local database <name> on <host>`, `hosted at <host>`, or
none without a config. Neither shows a key or password, and both go
through `redact`. Doctor's server line is scrubbed the same way.
Unshipped: the `store-line` variant appends "Store for this session:
{store}." to the shipped SessionStart line (the hook fills in the
session's bound store), and `lesson-store` pairs it with
`lesson-also-propose` as the eval condition. The eval's per-session
database names are now opaque hashes, since the store line shows the
agent that name.
Co-Authored-By: Claude Opus 5.5 <[email protected]>
Claude-Session: https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What changed
commons doctorhas a newstorecheck, right afterconfig. It showslocal database <name> on <host>orhosted at <host>[:port]. With no config it warnsnone until commons init (local) or commons join (hosted). Theserverline's detail now goes throughscrubtoo, because it used to print the server URL exactly as configured, password included.commons statusnow starts withstore: ..., printed before the database is reached. When the active store isn't the local database (hosted, or no config), a second line,statistics from: local database <name> on <host>, says where the figures come from:statusalways reads the local database.identity.store_label/identity.active_store, which use the newstore.connection.database_label. Only the host and database name are shown, never the key, the user or the password, and the result also goes throughredact(privacy.redaction.scrub).store-line: the shipped SessionStart start line plusStore for this session: {store}.. The hook fills{store}with the label of the session's bound store (cli/hooks.py_session_store, which picks the store the same waybackend_fordoes). The hook version hashes the template, so it doesn't change per machine.RELEASED_HOOK_TEXTSis unchanged (it gets the new entry only when this ships), and so arehook_texts.jsonandtest_hook_texts.py.lesson-store:lesson-also-propose's instructions plusstore-line's start line. Its instructions are 1,384 characters.tools/eval/runner.pydatabase_nameis now an opaque hash,commons_<16 hex>. The store line shows the agent its database's name, and the old name (commons_eval_<condition>_<index>_<scenario>) gave away the condition, the scenario (for exampleknown_config_default_wins) and the fact that this is an eval. This file wasn't on the ticket's list. The condition can't be run cleanly without this change.docs/eval/README.md: a short section on the two variants and the command below.Why
Ticket:
MIT-653. In two real misses, the agent used the wrong store or wrote the lesson to AGENTS.md instead of Commons. The owner sees the store indoctorandstatus(this ships). The agent would see it at session start (a draft, for the eval to judge).Not run (the eval budget is none, and the owner starts each batch). This is the command, on the four no-test scenarios plus the five controls, 3 runs each, on both models: 9 x 3 x 2 conditions x 2 models = 108 sessions.
(Run
reportagain with theclaude-opus-5-5conditions for Opus.)How it was tested
server/tests/test_active_store.py(new):doctor(the check and the printed line) andstatus(its first lines) for a local config, a hosted config and no config. The hosted server URL carries a password and the key looks like an Anthropic key, and neither appears in any output, nor does the database URL.store_labelfor a local database URL with a password, a hosted URL with user, password and port, an unreadable database URL, and a bare key passed as the server (redacted).store-lineandlesson-storefor local and hosted sessions, and underbaseline, which names no store.server/tests/test_variants.py:store-linediffers from the shipped text instart_lineonly.lesson-storehaslesson-also-propose's instructions andstore-line's start line, and resolves by name.RELEASED_HOOK_TEXTSis unchanged.server/tests/test_status.py: the summary test now expects thestore:line first, thenentries:.test_variants,test_status,test_hook_texts,test_hook_session_start,test_cli_utilities,test_mcp_instructions,test_eval_armsandtest_eval_store(223 passed). The full check passed in pre-commit (tools/check.py).Learned
commons statusalways reads the local database (connect()), even on a machine joined to a hosted server, so its figures there aren't the hosted store's. It now says so in astatistics from:line. Showing hosted figures would need a hosted status endpoint, which would be a ticket of its own.servercheck printed the configured URL verbatim. Any doctor or status text built from a URL should go throughprivacy.redaction.scrub.commons_<16 hex>(sha256 of condition, index and scenario), so a leftover database no longer says which session it came from. Look it up from the transcript stem instead. A--store NAMEexperiment store is shown to the agent under the name given, so pick a neutral one when running a store-line variant on it.🤖 Generated with Claude Code
https://claude.ai/code/session_01Qc1CQTb8r67J32HpkhS43y